Design of a Focused Crawler Based on Dynamic Computation of Topic Specific Weight Table
نویسندگان
چکیده
Abstract Focused Crawler aims to select relevant web pages from internet. These pages are relevant to some predefined topics. Previous focused crawlers have a problem of not keeping track of user interest and goals .The topic weight table is calculated only once statically and that is less sensitive to potential changes in environment. To address this problem we design a focused crawler based on dynamic computation of topic keywords and their weights. This weight table is constructed according to user query. To check the similarity of web page with respect to the topic keywords, a cosine similarity function is used and priority of extracted links is calculated.
منابع مشابه
Effective Focused Crawling Based on Content and Link Structure Analysis
A focused crawler traverses the web selecting out relevant pages to a predefined topic and neglecting those out of concern. While surfing the internet it is difficult to deal with irrelevant pages and to predict which links lead to quality pages. In this paper, a technique of effective focused crawling is implemented to improve the quality of web navigation. To check the similarity of web pages...
متن کاملPrioritize the ordering of URL queue in Focused crawler
The enormous growth of the World Wide Web in recent years has made it necessary to perform resource discovery efficiently. For a crawler it is not an simple task to download the domain specific web pages. This unfocused approach often shows undesired results. Therefore, several new ideas have been proposed, among them a key technique is focused crawling which is able to crawl particular topical...
متن کاملUnvisited URL Relevancy Calculation in Focused Crawling Based on Naïve Bayesian Classification
Vertical search engines use focused crawler as their key component and develop some specific algorithms to select web pages relevant to some pre-defined set of topics. Crawlers are software which can traverse the internet and retrieve web pages by hyperlinks. The focused crawler of a special-purpose search engine aims to selectively seek out pages that are relevant to a pre-defined set of topic...
متن کاملEmpirical evaluation of the link and content-based focused Treasure-Crawler
Indexing the Web is becoming a laborious task for search engines as the Web exponentially grows in size and distribution. Presently, the most effective known approach to overcome this problem is the use of focused crawlers. A focused crawler applies a proper algorithm in order to detect the pages on the Web that relate to its topic of interest. For this purpose we proposed a custom method that ...
متن کاملPDD Crawler: A focused web crawler using link and content analysis for relevance prediction
Majority of the computer or mobile phone enthusiasts make use of the web for searching activity. Web search engines are used for the searching; The results that the search engines get are provided to it by a software module known as the Web Crawler. The size of this web is increasing round-the-clock. The principal problem is to search this huge database for specific information. To state whethe...
متن کامل